Redundant Hardware System For Autonomous Vehicles

ABSTRACT

The technology relates to partially redundant equipment architectures for vehicles able to operate in an autonomous driving mode. Aspects of the technology employ fallback configurations, such as two or more fallback sensor configurations that provide some minimum amount of field of view (FOV) around the vehicle. For instance, different sensor arrangements are logically associated with different operating domains of the vehicle. Fallback configurations for computing resources and/or power resources are also provided. Each fallback configuration may have different reasons for being triggered, and may result in different types of fallback modes of operation. Triggering conditions may relate, e.g., to a type of failure, fault or other reduction in component capability, the current driving mode, environmental conditions in the vicinity of vehicle or along a planned route, or other factors. Fallback modes may involve altering a previously planned trajectory, altering vehicle speed, and/or altering a destination of the vehicle.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.16/215,713, filed Dec. 11, 2018, the entire disclosure of which isincorporated by reference herein. The present application is related toco-pending U.S. application Ser. No. 17/037,924, filed Sep. 30, 2020,which is a continuation of U.S. application Ser. No. 16/180,267,entitled Systems for Implementing Fallback Behaviors for AutonomousVehicles, filed Nov. 5, 2018 and issued as U.S. Pat. No. 10,838,417, theentire disclosures of which are incorporated by reference herein.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers or cargo fromone location to another. Such vehicles may operate in a fully autonomousmode or a partially autonomous mode where a person may provide somedriving input. In order to operate in an autonomous mode, the vehiclemay employ sensors, and use received sensor information to performvarious driving operations. However, if a sensor or other component ofthe system fails or otherwise suffers a degradation in capability, thismay adversely impact the driving capabilities of the vehicle.

BRIEF SUMMARY

The technology relates to redundant architectures for sensor, computeand power systems in vehicles configured to operate in fully orpartially autonomous driving modes. While it may be possible to havecomplete redundancy of every component and subsystem, this may not befeasible, especially with vehicles that have constraints on the size andplacement of sensor suites and other limiting factors such as cost.Thus, aspects of the technology employ fallback configurations forpartial redundancy. For instance, the fallback sensor configurations mayprovide some minimum amount of field of view (FOV) around the vehicle,as well as a minimum amount of computing power for perception andplanning processing.

According to aspects of the technology, a vehicle is configured tooperate in an autonomous driving mode. The vehicle comprises a drivingsystem, a perception system and a control system. The driving systemincludes a steering subsystem, an acceleration subsystem and adeceleration subsystem to control driving of the vehicle in theautonomous driving mode. The perception system has a plurality ofsensors configured to detect information about an environment around thevehicle. The plurality of sensors includes a first set of sensorsassociated with a first operating domain and a second set of sensorsassociated with a second operating domain. The control system isoperatively coupled to the driving system and the perception system. Thecontrol system includes a first computing subsystem associated with thefirst operating domain and a second computing subsystem associated withthe second operating domain. Each of the first and second computingsubsystems having one or more processors. The first and second computingsubsystems are each configured to receive sensor data from one or bothof the first set of sensors and the second set of sensors in a firstmode of operation. In response to the received sensor data in the firstmode of operation, the control system is configured to control thedriving system to drive the vehicle in the autonomous driving mode. Uponan error condition for one or more of the plurality of sensors, thefirst computing subsystem is configured to process sensor data from thefirst set of sensors in the first operating domain and the secondcomputing subsystem is configured to process sensor data from the secondset of sensors in the second operating domain. And in response to theerror condition, only one of the first computing subsystem or the secondcomputing subsystem is configured to control the driving system in afallback driving mode.

In an example, each of the first and second sets of sensors include atleast one sensor selected from the group consisting of lidar sensors,radar sensors, camera sensors, auditory sensors and positioning sensors.Here, each of the first and second sets of sensors may include arespective group of lidar, radar and camera sensors, each groupproviding a selected field of view of the environment around thevehicle.

Each of the sensors in the first set of sensors may have a respectivefield of view, each of the sensors in the second set of sensors may havea respective field of view, and in this case the respective fields ofview of the second set of sensors are different from the respectivefields of view of the first set of sensors. One or more interior sensorsmay be disposed in an interior of the vehicle. The one or more interiorsensors include at least one of a camera sensor, an auditory sensor andan infrared sensor.

In another example, the fallback driving mode includes a first fallbackmode and a second fallback mode. The first fallback mode includes afirst set of driving operations and the second fallback mode includes asecond set of driving operations different from the first set of drivingoperations. In this example, the first computing subsystem is configuredto control the driving system in the first fallback mode and the secondcomputing subsystem is configured to control the driving system in thesecond fallback mode.

In yet another example, the vehicle further comprises first and secondpower distribution subsystems. Here, in the fallback driving mode thefirst power distribution subsystem is associated with the firstoperating domain to power only devices of the first operating domain.Also in the fallback driving mode the second power distribution systemis associated with the second operating domain to power only devices ofthe second operating domain. And the first and second operating domainsare electrically isolated from one another. In this case, the firstpower distribution subsystem may be configured to provide power to afirst set of base vehicle loads in the first mode of operation and toprovide power to devices of the first operating domain in the fallbackdriving mode, and the second power distribution subsystem may beconfigured to provide power to a second set of base vehicle loadsdifferent than the first set of base vehicle loads in the first mode ofoperation and to provide power to devices of the second operating domainin the fallback driving mode.

In a further example, the plurality of sensors of the perception systeminclude a first set of fallback sensors operatively coupled to the firstcomputing subsystem, a second set of fallback sensors operativelycoupled to the second computing subsystem, and a set of non-fallbacksensors. In this scenario, the set of non-fallback sensors may beoperatively coupled to one or both of the first computing subsystem andthe second computing subsystem. And in yet another example, theplurality of sensors of the perception system include a subset ofsensors operatively coupled to both the first and second computingsubsystems in the fallback driving mode.

A method of operating a vehicle in an autonomous driving mode isprovided according to another aspect of the technology. The methodcomprises detecting, by a plurality of sensors of a perception system ofthe vehicle, information about an environment around the vehicle, theplurality of sensors including a first set of sensors associated with afirst operating domain and a second set of sensors associated with asecond operating domain; receiving, by a control system of the vehicle,the detected information about the environment around the vehicle assensor data, the control system including a first computing subsystemassociated with the first operating domain and a second computingsubsystem associated with the second operating domain; in response toreceiving the sensor data in a first mode of operation, the controlsystem controlling a driving system of the vehicle to drive the vehiclein the autonomous driving mode; detecting an error condition for one ormore of the plurality of sensors; upon detecting the error condition,the first computing subsystem processing sensor data from only the firstset of sensors in the first operating domain and the second computingsubsystem processing sensor data from only the second set of sensors inthe second operating domain; and in response to the error condition,only one of the first computing subsystem or the second computingsubsystem controlling the driving system in a fallback driving mode.

In one example, the fallback driving mode comprises a plurality offallback modes including a first fallback mode and a second fallbackmode. In this case, the first fallback mode may include a first set ofdriving operations and the second fallback mode may include a second setof driving operations different from the first set of drivingoperations. In this case, controlling the driving system in the fallbackdriving mode may include the first computing subsystem controlling thedriving system in the first fallback mode; or the second computingsubsystem controlling the driving system in the second fallback mode.

In another example, the method further comprises, in the fallbackdriving mode, powering, by a first power distribution subsystem of thevehicle, only devices of the first operating domain; and powering, by asecond power distribution subsystem of the vehicle, only devices of thesecond operating domain. In this scenario, the first power distributionsubsystem may provide power to a first set of base vehicle loads in thefirst mode of operation and power to devices of the first operatingdomain in the fallback driving mode; and the second power distributionsubsystem may provide power to a second set of base vehicle loadsdifferent than the first set of base vehicle loads in the first mode ofoperation and power to devices of the second operating domain in thefallback driving mode.

During the fallback driving mode, a subset of the plurality of sensorsof the perception system may provide sensor data to both the first andsecond computing subsystems. Controlling the driving system in thefallback driving mode may include at least one of altering a previouslyplanned trajectory of the vehicle, altering a speed of the vehicle, oraltering a destination of the vehicle. The method may also furthercomprise halting processing of sensor data from a non-fallback-criticalsensor during the fallback driving mode.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a perspective view an example vehicle configured foruse with aspects of the technology.

FIG. 2 illustrates a top view of the example vehicle of FIG. 1.

FIG. 3 is a block diagram of an example vehicle in accordance withaspects of the technology.

FIG. 4 is a block diagram of an example perception system in accordancewith aspects of the technology.

FIGS. 5A-B illustrate examples of regions around a vehicle in accordancewith aspects of the disclosure.

FIG. 6 illustrates example sensor fields of view in accordance withaspects of the disclosure.

FIG. 7 illustrates an example sensor assembly in accordance with aspectsof the disclosure.

FIG. 8 illustrates an example of sensor orientations in accordance withaspects of the disclosure.

FIGS. 9A-B illustrate examples of overlapping sensor fields of view inaccordance with aspects of the disclosure.

FIGS. 10A-B illustrate examples of operating in different domains inaccordance with aspects of the technology.

FIGS. 11A-D illustrate further examples of operating in differentdomains in accordance with aspects of the technology.

FIG. 12 illustrates an example self-driving system configuration inaccordance with aspects of the technology.

FIGS. 13A-B illustrate example power distribution configurations inaccordance with aspects of the technology.

FIG. 14 is a method of operation according to aspects of the technology.

DETAILED DESCRIPTION

The partially redundant vehicle architectures discussed herein areassociated with fallback configurations that employ different sensorarrangements that can be logically associated with different operatingdomains of the vehicle. Each fallback configuration may have differentreasons for being triggered, and may result in different types offallback modes of operation. Triggering conditions may relate, e.g., toa type of failure, fault or other reduction in component capability, thecurrent autonomous driving mode, environmental conditions in thevicinity of vehicle or along a planned route, or other factors.

Example Vehicle Systems

FIG. 1 illustrates a perspective view of a passenger vehicle 100, suchas a minivan, sedan or sport utility vehicle. FIG. 2 illustrates atop-down view of the passenger vehicle 100. The passenger vehicle 100may include various sensors for obtaining information about thevehicle's external environment. For instance, a roof-top housing 102 mayinclude a lidar sensor as well as various cameras, radar units, infraredand/or acoustical sensors. Housing 104, located at the front end ofvehicle 100, and housings 106 a, 106 b on the driver's and passenger'ssides of the vehicle may each incorporate a lidar or other sensor. Forexample, housing 106 a may be located in front of the driver's side dooralong a quarterpanel of the vehicle. As shown, the passenger vehicle 100also includes housings 108 a, 108 b for radar units, lidar and/orcameras also located towards the rear roof portion of the vehicle.Additional lidar, radar units and/or cameras (not shown) may be locatedat other places along the vehicle 100. For instance, arrow 110 indicatesthat a sensor unit (112 in FIG. 2) may be positioned along the read ofthe vehicle 100, such as on or adjacent to the bumper. And arrow 114indicates a series of sensor units 116 arranged along a forward-facingdirection of the vehicle. In some examples, the passenger vehicle 100also may include various sensors for obtaining information about thevehicle's interior spaces. The interior sensor(s) may include at leastone of a camera sensor, an auditory sensor and an infrared sensor.

While certain aspects of the disclosure may be particularly useful inconnection with specific types of vehicles, the vehicle may be any typeof vehicle including, but not limited to, cars, trucks, motorcycles,buses, recreational vehicles, etc.

FIG. 3 illustrates a block diagram 300 with various components andsystems of an exemplary vehicle configured to operate in a fully orsemi-autonomous mode of operation. By way of example, there aredifferent degrees of autonomy that may occur for a vehicle operating ina partially or fully autonomous driving mode. The U.S. National HighwayTraffic Safety Administration and the Society of Automotive Engineershave identified different levels to indicate how much, or how little,the vehicle controls the driving. For instance, Level 0 has noautomation and the driver makes all driving-related decisions. Thelowest semi-autonomous mode, Level 1, includes some drive assistancesuch as cruise control. Level 2 has partial automation of certaindriving operations, while Level 3 involves conditional automation thatcan enable a person in the driver's seat to take control as warranted.In contrast, Level 4 is a high automation level where the vehicle isable to drive without assistance in select conditions. And Level 5 is afully autonomous mode in which the vehicle is able to drive withoutassistance in all situations. The architectures, components, systems andmethods described herein can function in any of the semi orfully-autonomous modes, e.g., Levels 1-5, which are referred to hereinas “autonomous” driving modes. Thus, reference to an autonomous drivingmode includes both partial and full autonomy.

As illustrated in FIG. 3, the exemplary vehicle includes one or morecomputing devices 302, such as computing devices containing one or moreprocessors 304, memory 306 and other components typically present ingeneral purpose computing devices. The memory 306 stores informationaccessible by the one or more processors 304, including instructions 308and data 310 that may be executed or otherwise used by the processor(s)304. The computing system may control overall operation of the vehiclewhen operating in an autonomous mode.

The memory 306 stores information accessible by the processors 304,including instructions 308 and data 310 that may be executed orotherwise used by the processor 304. The memory 306 may be of any typecapable of storing information accessible by the processor, including acomputing device-readable medium. The memory is a non-transitory mediumsuch as a hard-drive, memory card, optical disk, solid-state, etc.Systems may include different combinations of the foregoing, wherebydifferent portions of the instructions and data are stored on differenttypes of media.

The instructions 308 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions”, “modules” and “programs” may be usedinterchangeably herein. The data 310 may be retrieved, stored ormodified by one or more processors 304 in accordance with theinstructions 308. In one example, some or all of the memory 306 may bean event data recorder or other secure data storage system configured tostore vehicle diagnostics and/or detected sensor data, which may be onboard the vehicle or remote, depending on the implementation.

The processors 304 may be any conventional processors, such ascommercially available CPUs. Alternatively, each processor may be adedicated device such as an ASIC or other hardware-based processor.Although FIG. 3 functionally illustrates the processors, memory, andother elements of computing devices 302 as being within the same block,such devices may actually include multiple processors, computingdevices, or memories that may or may not be stored within the samephysical housing. Similarly, the memory 306 may be a hard drive or otherstorage media located in a housing different from that of theprocessor(s) 304. Accordingly, references to a processor or computingdevice will be understood to include references to a collection ofprocessors or computing devices or memories that may or may not operatein parallel.

In one example, the computing devices 302 may form an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may capable of communicating with various components ofthe vehicle. For example, the computing devices 302 may be incommunication with various systems of the vehicle, including a drivingsystem including a deceleration system 312 (for controlling braking ofthe vehicle), acceleration system 314 (for controlling acceleration ofthe vehicle), steering system 316 (for controlling the orientation ofthe wheels and direction of the vehicle), signaling system 318 (forcontrolling turn signals), navigation system 320 (for navigating thevehicle to a location or around objects) and a positioning system 322(for determining the position of the vehicle). The autonomous drivingcomputing system may operate in part as a planner, in accordance withthe navigation system 320 and the positioning system 322, e.g., fordetermining a route from a starting point to a destination.

The computing devices 302 are also operatively coupled to a perceptionsystem 324 (for detecting objects in the vehicle's environment), a powersystem 326 (for example, a battery and/or gas or diesel powered engine)and a transmission system 330 in order to control the movement, speed,etc., of the vehicle in accordance with the instructions 308 of memory306 in an autonomous driving mode which does not require or needcontinuous or periodic input from a passenger of the vehicle. Some orall of the wheels/tires 328 are coupled to the transmission system 330,and the computing devices 302 may be able to receive information abouttire pressure, balance and other factors that may impact driving in anautonomous mode. The power system 326 may have multiple powerdistribution elements 327, each of which may be capable of supplyingpower to selected components and other systems of the vehicle.

The computing devices 302 may control the direction and speed of thevehicle by controlling various components. By way of example, computingdevices 302 may navigate the vehicle to a destination locationcompletely autonomously using data from the map information andnavigation system 320. Computing devices 302 may use the positioningsystem 322 to determine the vehicle's location and the perception system324 to detect and respond to objects when needed to reach the locationsafely. In order to do so, computing devices 302 may cause the vehicleto accelerate (e.g., by increasing fuel or other energy provided to theengine by acceleration system 314), decelerate (e.g., by decreasing thefuel supplied to the engine, changing gears, and/or by applying brakesby deceleration system 312), change direction (e.g., by turning thefront or other wheels of vehicle 100 by steering system 316), and signalsuch changes (e.g., by lighting turn signals of signaling system 318).Thus, the acceleration system 314 and deceleration system 312 may be apart of a drivetrain or other type of transmission system 330 thatincludes various components between an engine of the vehicle and thewheels of the vehicle. Again, by controlling these systems, computingdevices 302 may also control the transmission system 330 of the vehiclein order to maneuver the vehicle autonomously.

Navigation system 320 may be used by computing devices 302 in order todetermine and follow a route to a location. In this regard, thenavigation system 320 and/or memory 306 may store map information, e.g.,highly detailed maps that computing devices 302 can use to navigate orcontrol the vehicle. As an example, these maps may identify the shapeand elevation of roadways, lane markers, intersections, crosswalks,speed limits, traffic signal lights, buildings, signs, real time trafficinformation, vegetation, or other such objects and information. The lanemarkers may include features such as solid or broken double or singlelane lines, solid or broken lane lines, reflectors, etc. A given lanemay be associated with left and/or right lane lines or other lanemarkers that define the boundary of the lane. Thus, most lanes may bebounded by a left edge of one lane line and a right edge of another laneline.

The perception system 324 also includes sensors for detecting objectsexternal to the vehicle. The detected objects may be other vehicles,obstacles in the roadway, traffic signals, signs, trees, etc. As will bediscussed in more detail below, the perception system 324 is arranged tooperate with two (or more) sensor domains, such as sensor domain A andsensor domain B as illustrated. Within each domain, the system mayinclude one or both of an exterior sensor suite and an interior sensorsuite. As discussed further below, the exterior sensor suite employs oneor more sensors to detect objects and conditions in the environmentexternal to the vehicle. The interior sensor suite may employ one ormore other sensors to detect objects and conditions within the vehicle,such as in the passenger compartment.

FIG. 4 illustrates one example of the perception system 324. Forinstance, as shown each domain of the perception system 324 may includeone or more light detection and ranging (lidar) sensors 400, radar units402, cameras 404 (e.g., optical imaging devices, with or without aneutral-density filter (ND) filter), positioning sensors 406 (e.g.,gyroscopes, accelerometers and/or other inertial components), infraredsensors 408, acoustical sensors 410 (e.g., microphones or sonartransducers), and/or any other detection devices 412 that record datawhich may be processed by computing devices 302. The sensors of theperception system 324 in the exterior sensor suite may detect objectsoutside of the vehicle and their characteristics such as location,orientation, size, shape, type (for instance, vehicle, pedestrian,bicyclist, etc.), heading, and speed of movement, etc. The sensors ofthe interior sensor suite may detect objects within the vehicle (e.g.,person, pet, packages) as well as conditions within the vehicle (e.g.,temperature, humidity, etc.).

The raw data from the sensors and the aforementioned characteristics canbe processed by the perception system 324 and/or sent for furtherprocessing to the computing devices 302 periodically and continuously asthe data is generated by the perception system 324. Computing devices302 may use the positioning system 322 to determine the vehicle'slocation and perception system 324 to detect and respond to objects whenneeded to reach the location safely. In addition, the computing devices302 may perform calibration of individual sensors, all sensors in aparticular sensor assembly, or between sensors in different sensorassemblies or other physical housings.

As illustrated in FIGS. 1-2, certain sensors of the perception system324 may be incorporated into one or more sensor assemblies or housings.In one example, these may be arranged as sensor towers integrated intothe side-view mirrors on the vehicle. In another example, other sensorsmay be part of the roof-top housing 102. The computing devices 202 maycommunicate with the sensor assemblies located on or otherwisedistributed along the vehicle. Each assembly may have one or more typesof sensors such as those described above.

Returning to FIG. 3, computing devices 302 may include all of thecomponents normally used in connection with a computing device such asthe processor and memory described above as well as a user interfacesubsystem 334. The user interface subsystem 334 may include one or moreuser inputs 336 (e.g., a mouse, keyboard, touch screen and/ormicrophone) and one or more display devices 338 (e.g., a monitor havinga screen or any other electrical device that is operable to displayinformation). In this regard, an internal electronic display may belocated within a cabin of the vehicle (not shown) and may be used bycomputing devices 302 to provide information to passengers within thevehicle. Other output devices, such as speaker(s) 340 may also belocated within the passenger vehicle.

The passenger vehicle also includes a communication system 342. Forinstance, the communication system 342 may also include one or morewireless network connections to facilitate communication with othercomputing devices, such as passenger computing devices within thevehicle, and computing devices external to the vehicle such as inanother nearby vehicle on the roadway or a remote server system. Thenetwork connections may include short range communication protocols suchas Bluetooth, Bluetooth low energy (LE), cellular connections, as wellas various configurations and protocols including the Internet, WorldWide Web, intranets, virtual private networks, wide area networks, localnetworks, private networks using communication protocols proprietary toone or more companies, Ethernet, WiFi and HTTP, and various combinationsof the foregoing.

As illustrated in FIG. 3, the system may include one or more buses 344for transmitting information and/or power. The buses are able to providedirect or indirect connectivity between various components andsubsystems. For instance, a data communication bus may providebidirectional communication between cameras and other sensors of theperception system 324 and the computing devices 302. A power line may beconnected directly or indirectly to the power distribution elements 327of power system 326, or to a separate power source such as a batterycontrolled by the computing devices 302. Various protocols may beemployed for uni- or bi-directional dada communication. By way ofexample, a protocol using the Controller Area Network (CAN) busarchitecture, or an Ethernet-based technology such as 100Base-T1 (or1000Base-T1 or 10GBase-T) Ethernet may be employed. Other protocols suchas FlexRay can also be employed. Still further, an Automotive Audio Bus®(A2B) and/or other bus configurations may employed.

Example Implementations

In view of the structures and configurations described above andillustrated in the figures, various implementations will now bedescribed in accordance with aspects of the technology.

Partial Redundancy—Sensor Fallback Modes

The environment around the vehicle can be viewed as having differentquadrants or regions. One example of this is illustrated in FIG. 5A,which shows front, rear, right side and left side regions, as well asadjacent areas for the front right, front left, right rear and left rearareas around the vehicle. These regions are merely exemplary.

Various sensors may be located at different places around the vehicle(see FIGS. 1-2) to gather data from some or all of these regions. Forinstance, as seen in FIG. 5B, the three sensors 116 of FIG. 1 mayprimarily receive data from the front, front left and front rightregions around the vehicle.

Certain sensors may have different fields of view depending on theirplacement around the vehicle and the type of information they aredesigned to gather. For instance, different lidar sensors may be usedfor near (short range) detection of objects adjacent to the vehicle(e.g., less than 2-10 meters), while others may be used for far (longrange) detection of objects a hundred meters (or more or less) in frontof the vehicle. Mid-range lidars may also be employed. Multiple radarunits may be positioned toward the front, rear and/or sides of thevehicle for long-range object detection. And cameras may be arranged toprovide good visibility around the vehicle. Depending on theconfiguration, certain types of sensors may include multiple individualsensors with overlapping fields of view. Alternatively, other sensorsmay provide redundant 360° fields of view.

FIG. 6 provides one example 600 of sensor fields of view relating to thesensors illustrated in FIG. 2. Here, should the roof-top housing 102include a lidar sensor as well as various cameras, radar units, infraredand/or acoustical sensors, each of those sensors may have a differentfield of view. Thus, as shown, the lidar sensor may provide a 360° FOV602, while cameras arranged within the housing 102 may have individualFOVs 604. A sensor within housing 104 at the front end of the vehiclehas a forward facing FOV 606, while a sensor within housing 112 at therear end has a rearward facing FOV 608. The housings 106 a, 106 b on thedriver's and passenger's sides of the vehicle, respectively, may eachincorporate a lidar and/or other sensor having a respective FOV 610 a or610 b. Similarly, sensors within housings 108 a, 108 b located towardsthe rear roof portion of the vehicle each have a respective FOV 612 a or612 b. And the series of sensor units 116 arranged along aforward-facing direction of the vehicle may have respective FOVs 614,616 and 618. Each of these fields of view is merely exemplary and not toscale in terms of coverage range.

As noted above, multiple sensors may be arranged in a given housing oras an assembly. One example is shown in FIG. 7. This figure presents anexample 700 of a sensor assembly in accordance with aspects of thedisclosure. As shown, the sensor assembly includes a housing 702, whichis mounted to a portion 704 of a roof of the vehicle as shown by thedashed line. The housing 702 may be dome-shaped as shown, cylindrical,hemispherical, or have a different geometric shape. Within the housing702 is a first sensor 706 arranged remotely or away from the roof and asecond sensor 708 arranged closer to the roof. One or both of thesensors 706 and 708 may be LIDARs or other types of sensors. Disposedbetween the first sensor 706 and the second sensor 708 is an imagingassembly 710. The imaging assembly 710 includes one or more sets ofcameras arranged therealong. The housing 702 may be opticallytransparent at least along the places where the cameras are arranged.While not illustrated in FIG. 7, one or more processors, such asprocessors 304 of FIG. 3, may be included as part of the sensorassembly. The processors may be configured to process the raw imageryreceived from the various image sensors of the camera assembly, as wellas information received from the other sensors of the overall sensorassembly.

Depending on the configuration, various sensors may be arranged toprovide complementary and/or overlapping fields of view. For instance,the camera assembly 710 may include a first subsystem having multiplepairs of image sensors positioned to provide an overall 360° field ofview around the vehicle. The camera assembly 710 may also include asecond subsystem of image sensors generally facing toward the front ofthe vehicle, for instance to provide higher resolutions, differentexposures, different filters and/or other additional features, e.g., atan approximately 90° front field of view, e.g., to better identifyobjects on the road ahead. The field of view of this subsystem may alsobe larger or smaller than 90°, for instance between about 60-135°. FIG.8 provides an example 800 of the orientations of the various imagesensors of the first and second subsystems. The image sensors may beCMOS sensors, although CCD or other types of imaging elements may beemployed.

The elevation of the camera, lidar and/or other sensor subsystems willdepend on placement of the various sensors on the vehicle and the typeof vehicle. For instance, if the camera assembly 710 is mounted on orabove the roof of a large SUV, the elevation will typically be higherthan when the camera assembly is mounted on the roof of a sedan orsports car. Also, the visibility may not be equal around all areas ofthe vehicle due to placement and structural limitations. By varying thediameter of the camera assembly 710 and the placement on the vehicle, asuitable 360° field of view can be obtained. For instance, the diameterof the camera assembly 710 may vary from e.g., between 0.25 to 1.0meters, or more or less. The diameter may be selected to be larger orsmaller depending on the type of vehicle on which the camera assembly isto be placed, and the specific location it will be located on thevehicle.

As shown in FIG. 8, each image sensor pair of a first subsystem of thecamera assembly may include a first image sensor 802 and a second imagesensor 804. The first and second image sensors may be part of separatecamera elements, or may be included together in one camera module. Inthis scenario, the first image sensors 802 may be set to auto exposure,while the second image sensors 804 may be set to a fixed exposure, e.g.,using a dark or ND filter. As illustrated, 8 pairs of image sensors areshown, although more or fewer pairs may be employed. The second imagesubsystem includes image sensors 806, which may have a higher resolutionthan those of the first and second image sensors. This enhancedresolution may be particularly beneficial for cameras facing the frontof the vehicle, in order to provide the perception system 324 with asmuch detail of the scene in front of the vehicle as possible. FIG. 8illustrates 3 image sensors of the second subsystem, although more orfewer image sensors may be used. In the present example, 19 total imagesensors are incorporated into the camera assembly 710, including the 3from the second subsystem and 8 pairs from the first subsystem. Again,more or fewer image sensors may be employed in the camera assembly.

The exact field of view for each image sensor may vary, for instancedepending on features of the particular sensor. By way of example, theimage sensors 802 and 804 may have approximately 50° FOVs, e.g.,49°-51°, while the image sensors 806 may each have a FOV on the order of30° or slightly more, e.g., 5-10% more. This allows for overlap in theFOV for adjacent image sensors.

The selected amount of overlap is beneficial, as seams or gaps in theimagery or other data generated by the various sensors are undesirable.In addition, the selected overlap enables the processing system to avoidstitching images together. While image stitching may be done inconventional panoramic image processing, it can be computationallychallenging to do in a real-time situation where the vehicle isoperating in a self-driving mode. Reducing the amount of time andprocessing resources required greatly enhances the responsiveness of theperception system as the vehicle drives.

FIGS. 9A-B illustrate two examples of image sensor overlap betweenadjacent sensor regions. As shown, for image sensors covering the frontand front right regions around the vehicle, there may be an overlap 900(FIG. 9A) of between 0.5-5°, or alternatively no more than 6-10° ofoverlap. This overlap 900 may apply for one type of sensor, such asimage sensors 802 of FIG. 8. A different overlap may apply for anothertype of sensor. For instance, there may be an overlap 902 (FIG. 9B) ofbetween 2-8°, or alternatively no more than 8-12° of overlap for imagesensors 804 of FIG. 8. Larger or smaller overlaps may also be engineeredfor the system depending on, e.g., vehicle type, size, sensor type, etc.

Example Scenarios

As noted above, should a sensor or other component fail or encounter areduction in capability, it may limit the driving capabilities of thevehicle or prevent operation entirely. For instance, one or more sensorsmay encounter an error due to, e.g., a mechanical or electrical failure,degradation due to environmental conditions (such as extreme cold, snow,ice, mud or dust occlusion), or other factors. In such situations,fallback configurations are designed to provide at least a minimumamount of sensor information so that the perception and planning systemscan operate according to given operating mode. The operating mode mayinclude, e.g., completing a current driving activity (e.g., passengerdrop off at desired location) before being serviced, altering a routeand/or speed (e.g., exit a freeway and drive along surface streets,reduce speed to minimum posted limit for the route, etc.), or pullingover as soon as it is safe to do so.

By way of example, the fallback configurations may be associated withtwo distinct domains, e.g., domain A and domain B (see FIGS. 3-4). Eachdomain does not need to be a mirror image of the other. For example,certain sets of front-facing sensors may be allocated to domain A, whilea different set of front-facing sensors with different capabilities maybe allocated to domain B. In one scenario, each type of sensor may beincluded in each domain. In another scenario, at least one (set of)front-facing camera(s) is always available in each domain for trafficlight detection. In a further scenario, at least one sensor with a 360°field of view is allocated to each domain for detection andclassification of objects external to the vehicle. And in yet anotherscenario, one or more sensors may each be operatively part of bothdomains. Thus, there may be differences in capabilities based on whetherthe system is using the sensors of only domain A, only domain B, or acombination of domains A & B.

For instance, consider that in one scenario FIG. 6 illustrates astandard operating mode in which the various sensors from both (or all)domains are being used by the vehicle's perception and planning systems.FIG. 10A illustrates one example 1000 of domain A operation. And FIG.10B illustrates one example 1010 of domain B operation. Here, it can beseen that different sensors or groups of sensors may be used by onlyone, or by both domains. By way of example, the lidar sensor of FIG. 6may still provide a 360° FOV for both domains. However, in domain A itmay only provide ½ an amount of vertical resolution 1002 that it wouldduring standard operation. Similarly, in domain B the lidar sensor mayalso only provide ½ an amount of vertical resolution 1012 that it wouldduring standard operation.

And while the cameras within housing 102 (FIG. 2) have individual fieldsof view 604 that may also provide an overall 360° FOV, each domain mayemploy different camera sets. Here, for instance, domain A may includethe first image sensors 802 (e.g., auto exposure) to provide a firstimage sensing capability 1004, while domain B may include the secondimage sensors 804 (e.g., set to a fixed exposure) to provide a secondimage sensing capability 1014. However, both domain A and domain B mayinclude image sensors 806. This may be the case because image sensors806 may have a higher resolution than those of image sensors 802 and804, and face the front of the vehicle. This enhanced resolution may beparticularly beneficial for detecting street lights, pedestrians,bicyclists, etc. in front of the vehicle. Thus, as shown in both FIGS.10A and 10B, each domain may have image sensing capability 1006 fromsuch enhanced resolution image sensors. In other examples, domain B mayinclude image sensing capability 1006 where domain A, which includesother forward image sensing capabilities, does not.

FIGS. 11A-D illustrate how other sensors, such as radar sensors, can beemployed in different domains. FIG. 11A shows a combined FOV 1100 for aset of 6 sensors 1101—1106, having respective individual FOVs 1111-1116.The combined FOV 1100 may be available in a standard operating mode.During operation with domain A, as shown in FIG. 11B, only sensors 1101,1102, 1104 and 1106 are employed, resulting in a domain A FOVconfiguration 1110. And during operation with domain B, as shown in FIG.11C, only sensors 1101, 1103, 1105 and 1106 are employed, resulting in adomain B FOV 1120. In these examples for domains A and B, bothfront-facing sensors are utilized and provide overlapping fields ofview.

In this scenario, either the domain A or domain B configuration,individually, may provide sufficient sensor data for the vehicle tooperate in a first fallback mode. For instance, the vehicle may still beable to drive on the freeway, but in a slower lane or at a minimumposted speed or under another speed threshold. Or, the vehicle may beable to select an alternate route to the destination that has less turnsor fewer expected nearby objects (e.g., fewer cars), for instance bytaking surface streets as opposed to the freeway. In contrast, FIG. 11Dillustrates a different scenario 1120 for a second fallback mode, inwhich only front-facing sensors 1101 and 1106 are available. In thisfallback mode, the system may make significant changes to drivingoperations because only fields of view 1111 and 1116 are providingsensor input to the vehicle. This may include, for instance, selecting anearby drop-off point different than the planned destination, turning onthe hazard lights with the signaling system, etc.

While the above examples have discussed lidar, cameras and radar sensorsfor different fallback scenarios, other types of sensors, e.g.,positioning, acoustical, infrared, etc., may also be apportioned betweenthe different domains to provide partial redundancy sufficient tocontrol the vehicle in a given operating mode.

Partial Redundancy—Computer System Fallback Modes

Other aspects of the technology include redundancies that areincorporated into the computing system. During typical operation, afirst compute system (e.g., a planner subsystem) may generate atrajectory and send it to a second compute system in order to controlthe vehicle according to that trajectory. Each of these compute systemsmay be part of the computing devices 302 (FIG. 3). For redundancy, twosubsystems may each have one or more processors and associated memory.In this example, each subsystem is powered and operates independently,but shares sensor data and resources to handle basic vehicle operations.Should one of the subsystems fail, e.g., due to a CPU crash, kernelerror, power failure, etc., the other subsystem is capable ofcontrolling the vehicle in a designated fallback state of operation.

In one arrangement, both compute subsystems are capable of controllingthe vehicle in a standard operating mode as well as a fallback mode.Each subsystem is tied to a respective domain, e.g., via one or more CANbuses or FlexRay buses. However, the sensor suites in each domain do nothave to be identical, complementary or fully overlapping. For instance,certain “fallback” sensors may be assigned to control subsystem A (e.g.,sensors 1102, 1104 and 1106 of FIG. 11B), other fallback sensors may beassigned to control subsystem B (e.g., sensors 1101, 1103 and 1105 ofFIG. 11C), and other “non-fallback” sensors may be assigned only tocontrol subsystem A (see FIG. 12).

Unlike fallback sensors, non-fallback sensors need not be assigned toany particular domain or have any domain independence or redundancy.This is the case because non-fallback sensors are not related toproviding a minimum viable fallback capability, but rather may be usedfor a standard operating mode only. Nonetheless, if there is some otherreason (e.g., a vehicle integration) to put such sensors on one domainor the other, that can be accommodated without affecting fallbackoperation. One example is that one domain has more power supply headroomthan another, so the non-fallback sensors can be easily accommodated bythat domain. In another example, it may be easier to route wiring on oneparticular domain, so the non-fallback sensors could be tied to thatdomain.

FIG. 12 illustrates one example 1200 of a self-driving systemconfiguration. As shown, each domain includes one or more processors1202 a or 1202 b, which may be processors 304 from computing system 302.Each domain also include respective operating mode logic 1204 a, 1204 b,which may comprise software or firmware modules for execution by theprocessors 1202 a, 1202 b. The operating mode logic 1204 a, 1204 b mayinclude separate components for execution in a standard operating modeas well as one or more fallback operating modes. The fallback modes forthe different domains may involve operating the vehicle in differentways, for instance according to the types and capabilities of thefallback sensors associated with the respective domains. And as shown inFIG. 12, one of the domains (e.g., domain A) may include differentcompute resources, such as one or more graphical processing units (GPUs)1206 or other computational accelerator (e.g., an ASIC, FPGA, etc.).Here, one set of fallback sensors are associated with domain A, anotherset of fallback sensors is associated with domain B, and one or morenon-fallback sensors are also associated with domain A, although otherconfigurations are possible, for instance with the fallback sensorsbeing assigned to domain B, or in which one subset is assigned to domainA and another subset is assigned to domain B. The two (or more) computedomains do not need to have the same performance, or the same kinds ofcompute elements, although in certain configurations they may haveequivalent performance and/or the same kinds of compute elements.

By way of example only, in this configuration domain A may support theperception system (e.g., perception system 324 of FIGS. 3-4) duringstandard operation, while domain B may support the planner (e.g., routeplanning based on the navigation system 320 and the positioning system322 of FIG. 3) during standard operation. The GPU(s) 1206 may beconfigured to process sensor data received from some or all of theon-board sensors during standard operation.

In one scenario, each computing subsystem may have enough sensor inputand enough computing capability (e.g., processor and memory resources toprocess the received sensor data) to operate the vehicle in thecorresponding standard and fallback operating modes. The two (or more)domains may share information in the standard mode. This way, thevarious domains and subsystems can be used to provide optical FOVcoverage. According to aspects of the technology, some compute resourcesmay be held in reserve to handle fallback operation. And some typicaloperations in the standard mode may be stopped in case of fallback. Forexample, one or more forward-facing high resolution cameras (e.g.,cameras 806 of FIG. 8) may be considered non-fallback-critical sensors.As a result, if a fallback mode is entered, processing inputs from thesecameras may be halted.

Partial Redundancy—Power Distribution Fallback Modes

With regard to power distribution, aspects of the technology provide twofault independent power supplies, each having a battery backup. By wayof example, the dual power supplies are operationally isolated so thatmajor faults are limited to only one domain. Each power supply mayservice a particular set of base vehicle loads. FIG. 13A illustrates anexample redundant power distribution architecture 1300. As shown in thisscenario, each domain has its own independent power supply. Here, thepower supplies and loads of each domain are protected by an interveningelectronic fuse. In practice, during normal operation the e-fuse wouldbe a closed circuit. Should it detect a failure in the form of an overcurrent, under voltage or over temperature (or other aberrantcondition), it would quickly react to become an open-circuit, therebyisolating the two domains.

Each power supply has a battery backup, and is configured to provide anautomotive standard on the order of 12 volts (e.g., within the range of8-16 volts). Each domain has a separate power distribution block 1302 a,1302 b, such as power distribution elements 327 of FIG. 3.

The DC/DC unit is a voltage converter that that takes power from thehigh voltage battery pack upstream and converts it into low voltage(e.g., 12V) to charge 12V batteries. In the e-fused architecture, due tothe presence of the e-fuse, the DC/DC unit can charge both halves of thesystem when everything is working (non-faulted), so 2 DC/DC units arenot required. When a fault occurs, the system does not necessarily needto use the DC/DC unit because power can be supplied by the redundant lowvoltage backup batteries on each domain.

Another example of power redundancy that does not require an e-fusewould be a dual independent DC/DC system 1310 as shown in FIG. 13B.Here, each domain is served by its own DC power supply. In this example,each domain has a separate power distribution block 1312 a, 1312 b.Grounding of the system is provided so that return current paths for thedomains do not present single points of failure.

There may be fallback critical actuators, such as brakes, steeringand/or propulsion. And there may be non-fallback critical actuators,such as the horn, cabin lights, heating and air conditioning system,etc. Actuators that are considered fallback critical may be redundant(e.g., 2 or more separate actuators), and/or such actuators may receivepower from both domains. In contrast, non-fallback critical actuatorsmay have no redundant components and/or may only receive power from asingle domain.

There may also be redundancies in other subsystems of the vehicle. Forinstance, the braking and steering subsystems may be made redundant (inaddition to being powered redundantly). Likewise, communication withother vehicles or remote assistance may employ multiple cellular orother types of communication links. Two or more GPS receives may beused. Even wipers, sprayers or other cleaning components may configuredfor redundancy, and may be powered (as a base load) on one or both (ormore) of the domains. Base vehicle loads may include individual sensors,sensor suites, compute devices, actuators and/or other components, suchas those discussed with regard to FIG. 3.

FIG. 14 illustrates a method 1400 of operating a vehicle in accordancewith aspects of the technology. As shown in block 1402, informationabout the vehicle's environment is detected by the vehicle's sensors(e.g., lidar, radar, camera, auditory and/or positioning sensors). Atblock 1404, the detected information is received by a control system ofthe vehicle, such as computing system(s) 302 of FIG. 3 or system 1200 ofFIG. 12. Per block 1406, in response to receiving the sensor data in afirst mode of operation, a driving system of the vehicle is autonomouslycontrolled. At block 1408, an error condition of one or more of thevehicle's sensors is detected. This may be due, e.g., to a failure,fault or other reduction in sensor capability.

At block 1410, upon detecting the error condition, a first computingsubsystem processes sensor data from a first set of sensors in a firstoperating domain, and a second computing subsystem processes sensor datafrom a second set of sensors in a second operating domain. As a result,at block 1412, in response to the error condition one of the first orsecond computing subsystems controls the driving system of the vehiclein a fallback driving mode. In some examples, prior to detecting theerror condition, the first computing subsystem may be operating afunction (e.g., planner, perception, etc.) in a standard operating mode,and the second computing subsystem may be operating another function inthe standard operating mode.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements. Theprocesses or other operations may be performed in a different order orsimultaneously, unless expressly indicated otherwise herein.

1. A vehicle configured to operate in an autonomous driving mode, thevehicle comprising: a driving system configured to perform drivingactions of the vehicle; a perception system having a plurality ofsensors configured to detect information about an environment around thevehicle; and a control system operatively coupled to the driving systemand the perception system, the control system including a firstcomputing subsystem associated with a first operating domain and asecond computing subsystem associated with a second operating domaindifferent from the first operating domain, each of the first and secondcomputing subsystems having one or more processors, wherein: the controlsystem is configured to receive sensor data from the perception system,and in response to the received sensor data the control system isconfigured to control the driving system to perform the driving actionsto drive the vehicle in the autonomous driving mode; the control systemis configured to identify an error condition of the vehicle; uponidentification of the error condition, the first computing subsystem isconfigured to process sensor data received from only a first set of theplurality of sensors according to the first operating domain and thesecond computing subsystem is configured to process sensor data receivedfrom only a second set of the plurality of sensors according to thesecond operating domain, the second set of sensors being different fromthe first set of sensors; and only one of the first computing subsystemor the second computing subsystem is configured to control the drivingsystem in a fallback driving mode based on the error condition.
 2. Thevehicle of claim 1, wherein each of the first and second sets of sensorsincludes at least one sensor selected from the group consisting of lidarsensors, radar sensors, camera sensors, auditory sensors and positioningsensors.
 3. The vehicle of claim 2, wherein each of the first and secondsets of sensors includes a respective group of lidar, radar and camerasensors, each group providing a selected field of view of theenvironment around the vehicle.
 4. The vehicle of claim 1, wherein eachof the sensors in the first set of sensors has a respective field ofview, each of the sensors in the second set of sensors has a respectivefield of view, and the respective fields of view of the second set ofsensors are different from the respective fields of view of the firstset of sensors.
 5. The vehicle of claim 1, wherein the perception systemfurther includes one or more interior sensors disposed in an interior ofthe vehicle, the one or more interior sensors including at least one ofa camera sensor, an auditory sensor and an infrared sensor.
 6. Thevehicle of claim 5, wherein the first computing subsystem is furtherconfigured to process sensor data received from the one or more interiorsensors according to the first operating domain.
 7. The vehicle of claim1, wherein the plurality of sensors of the perception system includes atleast one common sensor operatively coupled to both the first and secondcomputing subsystems for operation of the vehicle in the fallbackdriving mode.
 8. The vehicle of claim 1, wherein: the fallback drivingmode includes a first fallback mode and a second fallback mode, thefirst fallback mode implementing a first set of driving operations andthe second fallback mode implementing a second set of driving operationsdifferent from the first set of driving operations; the first computingsubsystem is configured to control the driving system in the firstfallback mode; and the second computing subsystem is configured tocontrol the driving system in the second fallback mode.
 9. The vehicleof claim 8, wherein one of the first fallback mode or the secondfallback mode operates based on sensor data received from onlyfront-facing sensors of the perception system.
 10. The vehicle of claim8, where each of the first and second fallback modes involves operatingthe vehicle in different ways according to types of the sensors andcapabilities of the sensors associated with the respective first andsecond operating domain.
 11. The vehicle of claim 1, wherein thefallback driving mode supports continuing driving operations along aroute towards a planned destination with at least one of an alteredvehicle speed, actuation of hazard lights, or selecting a drop-off pointdifferent than the planned destination.
 12. The vehicle of claim 1,wherein the fallback driving mode supports completing a current drivingactivity of the vehicle before having the vehicle serviced.
 13. Thevehicle of claim 1, wherein the fallback driving mode supports alteringa route of the vehicle or pulling over upon determination that it issafe to pull over.
 14. The vehicle of claim 1, further comprising firstand second power distribution subsystems, wherein in the fallbackdriving mode: the first power distribution subsystem is associated withthe first operating domain to power only devices of the first operatingdomain; and the second power distribution subsystem is associated withthe second operating domain to power only devices of the secondoperating domain.
 15. A method of operating a vehicle in an autonomousdriving mode, the method comprising: detecting, by a plurality ofsensors of a perception system of the vehicle, information about anenvironment around the vehicle, the plurality of sensors including afirst set of sensors associated with a first operating domain and asecond set of sensors associated with a second operating domain, thesecond set of sensors being different from the first set of sensors;receiving, by a control system of the vehicle, the detected informationabout the environment around the vehicle as sensor data, the controlsystem including a first computing subsystem associated with the firstoperating domain and a second computing subsystem associated with thesecond operating domain; controlling, by the control system, a drivingsystem of the vehicle in order to perform driving actions to drive thevehicle in the autonomous driving mode; identifying, by the controlsystem, an error condition of the vehicle; and upon identifying theerror condition: processing sensor data by the first computing subsystemthat is received from only the first set of sensors according to thefirst operating domain, processing sensor data by the second computingsubsystem that is received from only the second set of sensors accordingto the second operating domain; and only one of the first computingsubsystem or the second computing subsystem controlling the drivingsystem in a fallback driving mode based on the error condition.
 16. Themethod of claim 15, wherein the fallback driving mode comprises aplurality of fallback modes including a first fallback mode and a secondfallback mode.
 17. The method of claim 16, wherein the first fallbackmode includes a first set of vehicle control operations and the secondfallback mode including a second set of vehicle control operationsdifferent from the first set of vehicle control operations.
 18. Themethod of claim 15, wherein the perception system further includes oneor more interior sensors disposed in an interior of the vehicle and themethod further includes processing, by the first computing subsystem,sensor data received from the one or more interior sensors according tothe first operating domain.
 19. The method of claim 15, wherein thefallback driving mode supports continuing driving operations along aroute towards a planned destination with at least one of an alteredvehicle speed, actuation of hazard lights, or selecting a drop-off pointdifferent than the planned destination.
 20. The method of claim 15,wherein the fallback driving mode supports one of: completing a currentdriving activity of the vehicle before having the vehicle serviced;altering a route of the vehicle; or pulling over upon determination thatit is safe to pull over.